Validation

This document lays out results for estimating statistical power and minimum detectable effect size (MDES) for: Blocked RCT, with 2 levels, and randomization done at level 1 (individual level). This document contains models assuming constant treatment effects, fixed treatment effects, and random treatment effects.

We compare results of the derived methods against Monte Carlo Simulations & the PowerUp R package on power.

Power Validation

In this section, we validate power results for different definitions of power and different adjustment procedures.

Default parameters:

M = 3
J = 20
rho: \(\rho\) = 0.5
ATE ES = 0.125, 0.125, 0.125
R2.1: \(R^2_1\) = 0, 0, 0

Parameters by model type:

Omega: \(\omega_2 = 0\) for constant effects, \(\omega_2\) = for fixed and random
ICC: \(\text{ICC}_2 = 0\) for constant and fixed effects, \(\text{ICC}_2\) = 0.2, 0.2, 0.2 for random effects

Varying school size

\(\bar{n} = 100\)

\(\bar{n} = 75\)

\(\bar{n} = 50\)

Varying R2

\(R^2_1 =\) 0.6, 0.6, 0.6

\(R^2_2 =\) 0.6, 0.6, 0.6 (This should not affect power!)

Varying rho

\(\rho =\) 0.2

\(\rho =\) 0.8

Varying true positives

ATE (ES) = 0.125, 0, 0

Varying ICC

\(\text{ICC}_2\) = 0.8, 0.8, 0.8

MDES validation

In this section, we validate the derived minimum detectable effect size (MDES) function across different MTP (Bonferroni, Benjamini-Hocheberg and Holms) and power definition (individual and minimum_1). The validation is carried out by generating power values for the specific MTP and power definitions for a specific MDES. Then, we supply the generated estimate power as an input parameter to see if the specified MDES is returned.

Power: 0.65 - 0.87

Below, we have input parameters of number of outcomes, minimum detectable effect size, statistical power, and significance level that would estimate a high statistical power value of above 0.65 to 1.00 for different definitions of power. This estimated power value is used as an input to the mdes function where we validate if the returned mdes value is close to the mdes value that was inputted to provide the estimated power.

Individual Power

School size: 50

The Bonferroni, Benjamini Hocheberg and Holms’ adjusted MDES and the targeted MDES are within an acceptable margin of error for the specified targetted power. The adjusted MDES is estimated at the corresponding individual power in the table below. The corresponding individual power is the power at which our MDES method stops looking for a MDES value close to the target MDES.

Sample size validation